A Record Linkage-Based Data Deduplication Framework with DataCleaner Extension

The data management process is characterised by a set of tasks where quality (DQM) one the core components. Data quality, however, multidimensional concept, nature issues very diverse. One most widely anticipated challenges, which becomes particularly vital when come from multiple sources typical situation in current data-driven world, duplicates or non-uniqueness. Even more, were recognised to be key domain-specific dimensions context Internet Things (IoT) application domains, smart grids and health dominate most. Duplicate lead inaccurate analyses, leading wrong decisions, negatively affect and/or processing activities such as development models, forecasts, simulations, have negative impact on customer service, risk crisis management, service personalisation terms both their accuracy trustworthiness, decrease user adoption satisfaction, etc. determination elimination known deduplication, while finding more databases that refer same entities Record Linkage. To find duplicates, sets are compared with each other using similarity functions usually used compare two input strings similarities between them, requires quadratic time complexity. defuse complexity problem, especially large sources, record linkage methods, blocking sorted neighbourhood, used. In this paper, we propose six-step deduplication framework. operation framework demonstrated simplified example research artifacts, publications, projects others real-world institution representing Research Information Systems (RIS) domain. make proposed usable integrated it into tool already practice, developing prototype an extension for well-known DataCleaner. detects visualises thereby identifying providing identified redundancies user-friendly manner allowing further elimination. By removing redundancies, improved therefore improving analyses decision-making. This study makes call researchers take step towards “golden record” can achieved all resolved, thus moving absolute quality.